corpus: harvest realized-outcome verifier cases - #3402

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Workflow source detected

PR #3402 now has valid workflow source context (origin=github_issue ref=#2819).

A linked GitHub issue is present for this PR.

_{Comment @coderabbitai help to get the list of available commands.}

Automated Status Summary

Head SHA: df0b1f5
Latest Runs: ⏳ pending — Gate
Required contexts: summary
Required: core tests (3.12): ⏳ pending, core tests (3.13): ⏳ pending, docker smoke: ⏳ pending, gate: ⏳ pending

Coverage Overview

• Coverage history entries: 0

Updated automatically; will refresh on subsequent CI/Docker completions.

Keepalive checklist

Scope

No scope information available

Tasks

• No tasks defined

Acceptance criteria

• No acceptance criteria defined

Orphan stewardship handoff: this harvest follows closed design/source #2819, and explicit source metadata plus agent:codex / agents:keepalive / autofix routing are now present. The complete diff adds 14 staging records (9 Workflows, 1 Trend_Model_Project, 4 Fine-Art-Archive); the PR description now states the correct count and staging-only scope.

Receiving worker: Reviewed Repo Merge Verify Closer (imi-merge-verify-closer), verified ACTIVE hourly at minute 20. Concrete next action, due 2026-09-07T13:20:00Z: verify the metadata corrections against exact head 7f88e1b, obtain disposition of active threads PRRT_kwDOQprj9M6fzEV2 and PRRT_kwDOQprj9M6fzE0Z, and apply normal unchanged-head, required-check and zero-active-thread merge gates. Reviewed Repo Backlog Opener remains ACTIVE hourly if bounded branch repair is needed. No source commit or check rerun was made by this steward; #2819 remains closed.

Runner dispatch state for codex on PR #3402. Do not edit.

🤖 Keepalive Loop Status

PR #3402 | Agent: Codex | Iteration 0/12

Current State

🔍 Failure Classification

| Error type | infrastructure |
| Error category | unknown |
| Suggested recovery | Capture logs and context; retry once and escalate if the issue persists. |

🤖 Keepalive Loop Status

PR #3402 | Agent: Codex | Iteration 2/12

Current State

🔍 Failure Classification

| Error type | infrastructure |
| Error category | unknown |
| Suggested recovery | Capture logs and context; retry once and escalate if the issue persists. |

🤖 Bot Comment Handler

• Agent: codex
• Bot comments to address: 1
• Exact PR head: 7f88e1b
• Controller part: 1 of 1

The agent is reassigned only after every controller part is durable on the PR.
Each entry links to the authoritative review thread containing its full context.

Active thread controller

• PRRT_kwDOQprj9M6fzE0Z — config/model_eval_corpus_staging.json:1290
  • corpus: harvest realized-outcome verifier cases #3402 (comment)
  • Acceptance criterion: The PR description says this change contains high-confidence realized-outcome cases and that ambiguous cases were routed to the auto-expiring staging file “not here”, but the only file modified is the staging corpus and the new entries are all clean-pass harvested cases. Per tools/harvest_verifier_corpus.py, this file is specifically for l...

Required outcome

1. Inspect every listed active thread on the exact head.
2. Implement and validate any still-valid criterion; do not make no-op edits.
3. Reply with exact-head evidence and request a thread-specific reviewer disposition.
4. Never self-resolve reviewer threads.
5. Do not report completion while any listed thread remains active; a generic top-level review is insufficient.

Closer review-thread disposition (cursor closer)

Audited the harvest diff and tools/harvest_verifier_corpus.py routing on exact head after rebasing onto current main.

CodeRabbit count mismatch (14 vs 18)

The commit adds 14 new staging records (9× stranske/Workflows, 1× stranske/Trend_Model_Project, 4× stranske/Fine-Art-Archive). The walkthrough’s “18” figure was incorrect; there are no missing records in this diff. Case IDs added: workflows-2981…workflows-2993, trend_model_project-5799, fine-art-archive-470…fine-art-archive-474.

Copilot staging vs frozen-corpus concern

This PR is staging-only by design for this harvest run. partition() routes clean-pass cases with confidence: low (merged younger than stability_days) into the auto-expiring staging file; only confidence: high cases promote into config/model_eval_pilot.json. A fresh dry-run on this head reports 0 promoted, 156 staged, and this PR intentionally contains no pilot-corpus edits because none of the newly harvested cases cleared the stability window yet.

Rebased branch onto current main (f294b789); fresh CI required before merge. Both threads are dispositioned as satisfied with this evidence.

Absent-check record (pre-merge)

gh pr checks lists what did report, so a check that never started is missing from the list, not red. Running the absent-check comparison by hand for this head (the check_checks_reported.py reporter is hardcoded to stranske/Orchestrator and does not cover this repo):

• Reported on head f294b789: 35 check-runs + 2 commit statuses (Gate / gate success, CodeRabbit success). 26 pass / 15 skipping / 0 fail / 0 pending.
• Reference set: check-run names reporting on ≥6 of the last 8 merged PRs (chore(main): release 1.32.2 #3400, fix(gate): survive a fork PR's read-only token when writing the Gate status #3398, chore(main): release 1.32.1 #3397, fix(repo-review): restart repo analysis on head drift #3396, fix(repo-review): fail closed on incomplete docs drift #3395, fix(sync): preserve current Claude review action pin #3393, config(models): refresh stale OpenAI pricing and record the Astra CLI-version constraint #3390, fix(sync): renew the sealed delivery Gate trigger #3388) → 36 names.
• Absent on this head: gate and auto-pilot — 8/8 on the reference set, 0 here.

Disposition — structural, not a verification gap. Both names are jobs of Agents Auto-Pilot (.github/workflows/agents-auto-pilot.yml). Its pull_request trigger is types: [labeled, closed] — it does not subscribe to synchronize. This head was produced by the rebase force-push, not by a label event, so those two jobs could not have run on it. The reference PRs carry them because a label event fired on their final head. No workflow was deleted, held, cancelled or path-filtered away, and neither name is a required check — branch protection requires summary, which reported and passed (run 34134676295).

Confirmed on head f294b789 by workflow run: Gate, Agents PR meta manager, Health 40/44/45/50/52, Maint 52, PR 11, Selftest CI all completed success; PR 46 Dependency Repair Contract completed skipped. Agents Auto-Pilot has no run on this SHA, as its trigger set predicts.

Merging on that basis. Zero active non-outdated review threads; MERGEABLE/CLEAN; post-push review window elapsed (push 14:45Z, head unchanged on re-read).

✅ Progress Review (Round 5)

Recommendation: CONTINUE
Alignment Score: 10.0/10

Feedback

Work appears aligned. Continue toward task completion.

This review was triggered because the agent has been working for 5 rounds without completing any task checkboxes.
The review evaluates whether recent work is advancing toward the acceptance criteria.

Provider Comparison Report

Provider Summary

Agreement

• Verdict: FAIL (all providers)
• Correctness: scores within 1 point (avg 3.5/10, range 3.0-4.0)
• Completeness: scores within 1 point (avg 1.0/10, range 1.0-1.0)
• Testing: scores within 1 point (avg 0.5/10, range 0.0-1.0)
• Risks: scores within 1 point (avg 3.5/10, range 3.0-4.0)

Disagreement

Unique Insights

• openai: The change only appends static entries to config/model_eval_corpus_staging.json. It does not implement or modify the required automatic join between live pr_verifier decisions and realized PR outcomes in tools/harvest_verifier_corpus.py or maint-79-verifier-corpus-harvest.yml.; None of the required workflow behavior is addressed: no freshness-triggered pilot dispatch, candidate injection into the pilot, tiered promotion handling, rollback metadata, or quality_gate_breach-triggered reversion is implemented in this diff.; The required promotion-policy behavior cannot be established from these data-only changes. There are no changes to tools/prepare_model_promotion.py, maint-86-model-promotion-prepare.yml, or policy-preserving enforcement logic.; No regression tests were added or updated in the three explicitly required test files. In particular, this diff does not demonstrate N simulated decisions/outcomes yielding N correctly labeled corpus cases, auto-promotion for qualifying same-family/cost-nonincreasing candidates, approval gating for risky swaps, or rollback behavior.; The appended cases contain stable-looking case_id values, but only record PASS/clean-pass labels and do not include verifiable decision/outcome linkage metadata in the change. The diff therefore provides no evidence that these labels were automatically derived from realized outcomes rather than manually curated.
• anthropic: The diff only adds static JSON entries to config/model_eval_corpus_staging.json (126 lines of data); it contains no changes to tools/harvest_verifier_corpus.py, tools/prepare_model_promotion.py, or any of the referenced workflow files (maint-77, maint-79, maint-86).; None of the required test files (tests/tools/test_harvest_verifier_corpus.py, tests/tools/test_prepare_model_promotion.py, tests/workflows/test_model_eval_pilot_workflow.py) were added or modified, so the acceptance criterion requiring these tests to pass with non-zero collection cannot be verified from this PR.; The acceptance criteria describe substantial new automation behavior (auto-dispatch pilot on new catalog candidate, auto-harvesting corpus with stable case identities, tiered auto-promotion vs approval-gated PRs, quality_gate_breach auto-revert) — none of this logic is present in the diff.; The PR title/scope ('harvest realized-outcome verifier cases') suggests this is meant to be a narrow follow-up populating corpus data, but it is being evaluated against a much broader design issue's acceptance criteria which it does not satisfy.; CI success reported is generic (lint, JS tests, Python tests) and does not confirm the specific pytest invocation named in the acceptance criteria actually exercises new functionality, since no corresponding code changes exist.; Risk: harvested data entries are unverifiable from the diff alone (no code to establish provenance/join logic), raising questions about how these case entries were generated without corresponding tool changes in this PR.

🔍 LangSmith Traces

• openai
• anthropic

Closer verifier disposition — FAIL is a false positive caused by a wrong source binding on this PR

Provider Comparison (2026-09-07T15:03:49Z) returned FAIL from both providers (openai 95%, anthropic 85%). Both graded this PR against issue #2819's five tasks and six acceptance criteria — workflow dispatch logic, promotion tiering, rollback metadata, three named test files — and correctly observed that a 126-line append to config/model_eval_corpus_staging.json satisfies none of them.

The PR was never supposed to be graded against #2819. This is the weekly maint-79-verifier-corpus-harvest.yml job on branch verifier-corpus-harvest/auto. Its only job is to append realized-outcome cases to the staging corpus. #2819's implementation landed on 2026-07-25 (#2831, #2832) and the issue was CLOSED on 2026-08-15, three weeks before this PR opened. The verifier is holding a recurring data job to the acceptance criteria of a finished design epic.

The binding is new and is a regression. The six prior harvest PRs on this same branch carry no meta:issue marker and no closing reference:

The workflow's own body text (maint-79-verifier-corpus-harvest.yml:73) says only "Automated corpus growth (#2819 move 2)" — a provenance mention in owner/repo#N form, which source_context.js:206-212 explicitly skips (token.includes('/'), and the preceding character is a word character). Neither that workflow nor source_context.js / agents_pr_meta_update_body.js / agents-pr-meta-v4.yml has changed since 2026-08-24, so the mechanism is not yet identified.

Disposition: no code remedy on this PR, no follow-up PR against #2819. The merged content is correct — 14 harvested clean-pass cases, all expected_verdict: PASS, provenance: harvested, consistent with the 1,279 lines already in the file. #2819 stays closed; it is not reopened on the strength of a FAIL that was measured against the wrong contract. Merging did not misclose anything because #2819 was already closed — but the identical binding on an open epic would have closed it on a JSON append, which is why this is filed rather than just annotated: see #3407.

Closer lane, 2026-09-07. Verified against the merged diff, the six prior harvest PRs, issue #2819's state, and the commit history of all four pr-meta binding files.